Image sensing apparatus, control method thereof, and program for suppressing image deterioration caused by foreign substance

ABSTRACT

An image sensing apparatus includes an image sensing unit having an image sensor, an optical member which is arranged in front of the image sensor, a foreign substance detection unit which detects, from a foreign substance detection image including the image of a foreign substance adhered to the surface of the optical member, a recording unit which, when shooting a moving image, records moving image data generated based on image signals successively output from the image sensing unit, and records foreign substance information and lens information in addition to the moving image data, and a lens information obtaining unit which, when the lens information is updated by operating the imaging lens by a user during moving image shooting, obtains the updated lens information. When the lens information obtaining unit obtains the updated lens information, the recording unit records the updated lens information in addition to the moving image data.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique of suppressing imagedeterioration caused by a foreign substance adhered to the surface of anoptical low-pass filter or the like in an image sensing apparatus usingan image sensor such as a CCD sensor or CMOS sensor.

2. Description of the Related Art

In a conventional lens-interchangeable digital camera, a foreignsubstance (to be simply referred to as dust hereinafter) such as dust ormote is sometimes adhered to an optical system or the surface of animage sensor cover glass or optical filter arranged in front of an imagesensor (which will be generically referred to as an image sensor opticalcomponent). When dust is adhered to the image sensor optical component,it blocks light, and an image at the light-blocked portion is not shot,degrading the quality of the shot image.

Such dust on the image sensor is generally adhered not to the surface ofthe image sensor but to the surface of the cover glass or opticalfilter. The imaging state changes depending on the aperture value orpupil position of the imaging lens. More specifically, when the aperturevalue is almost the full-aperture one, the dust image blurs, and even ifsmall dust is adhered, it does not matter. When the aperture value islarge, a sharp dust image is formed, and even small dust adverselyaffects the entire image. To solve this problem, there is known a methodof making dust less noticeable. According to this method, an image ofonly dust on an image sensor is prepared in advance by shooting a whitewall or the like while setting the lens to a large aperture value. Thisimage is used in combination with a shot still image (see JapanesePatent Laid-Open No. 2004-222231).

There have recently been proposed digital cameras with a moving imageshooting function in addition to a still image shooting function. When alens-interchangeable digital camera has the moving image shootingfunction, the aperture value and pupil position of the imaging lenschange during moving image shooting in accordance with a lens operation(e.g., zoom operation). As a result, the imaging state of dust on theimage sensor changes in every frame of the moving image. It is necessaryto pay attention to the correspondence between an image shot for dustdetection and actually shot images associated with it. If the methoddescribed in Japanese Patent Laid-Open No. 2004-222231 is simply appliedto moving image shooting, the correspondence must be cumbersomelychecked for each frame of the moving image.

SUMMARY OF THE INVENTION

The present invention is made to overcome the conventional drawbacks,and suppresses the influence, on a moving image, of a foreign substancesuch as dust adhered to a cover glass, filter, or the like arranged infront of an image sensor.

According to the first aspect of the present invention, there isprovided an image sensing apparatus comprising an image sensing unithaving an image sensor which photoelectrically converts an object imageformed via an imaging lens, an optical member which is arranged in frontof the image sensor, a foreign substance detection unit which detects,from a foreign substance detection image including an image of a foreignsubstance adhered to a surface of the optical member, foreign substanceinformation serving as information including information on at least aposition and size of the foreign substance, a recording unit which, whenshooting a moving image, records moving image data generated based onimage signals successively output from the image sensing unit, andrecords, in addition to the moving image data, lens informationincluding the foreign substance information, information of an aperturevalue of the imaging lens, and information of a pupil position, and alens information obtaining unit which, when the lens information isupdated by operating the imaging lens by a user during moving imageshooting, obtains the updated lens information, wherein when the lensinformation obtaining unit obtains the updated lens information, therecording unit records the updated lens information in addition to themoving image data.

According to the second aspect of the present invention, there isprovided a method of controlling an image sensing apparatus including animage sensing unit having an image sensor which photoelectricallyconverts an object image formed via an imaging lens, and an opticalmember which is arranged in front of the image sensor, the methodcomprising a foreign substance detection step of detecting, from aforeign substance detection image including an image of a foreignsubstance adhered to a surface of the optical member, foreign substanceinformation serving as information including information on at least aposition and size of the foreign substance, a recording step of, whenshooting a moving image, recording moving image data generated based onimage signals successively output from the image sensing unit, andrecording, in addition to the moving image data, lens informationincluding the foreign substance information, information of an aperturevalue of the imaging lens, and information of a pupil position, and alens information obtaining step of, when the lens information is updatedby operating the imaging lens by a user during moving image shooting,obtaining the updated lens information, wherein in the recording step,when the updated lens information is obtained in the lens informationobtaining step, the updated lens information is recorded in addition tothe moving image data.

According to the third aspect of the present invention, there isprovided an image sensing apparatus comprising an image sensing unitwhich photoelectrically converts an object image to generate an imagesignal, a foreign substance detection unit which detects, from a foreignsubstance detection image signal obtained by the image sensing unit,foreign substance information serving as information on at least aposition and size of the foreign substance in an image sensing frame ofthe image sensing unit, a lens information obtaining unit which obtainslens information of a lens used to image an object, and a recording unitwhich, when shooting a moving image, records moving image data generatedbased on image signals successively output from the image sensing unit,and records, in addition to the moving image data, the foreign substanceinformation detected by the foreign substance detection unit and thelens information obtained by the lens information obtaining unit,wherein the recording unit fragments the moving image data, records thefragments, adds lens information obtained by the lens informationobtaining unit to each fragment, and records the lens information.

According to the fourth aspect of the present invention, there isprovided a method of controlling an image sensing apparatus having animage sensing unit which photoelectrically converts an object image togenerate an image signal, the method comprising a foreign substancedetection step of detecting, from a foreign substance detection imagesignal obtained by the image sensing unit, foreign substance informationserving as information on at least a position and size of the foreignsubstance in an image sensing frame of the image sensing unit, a lensinformation obtaining step of obtaining lens information of a lens usedto image an object, and a recording step of, when shooting a movingimage, recording moving image data generated based on image signalssuccessively output from the image sensing unit, and recording, inaddition to the moving image data, the foreign substance informationdetected in the foreign substance detection step and the lensinformation obtained in the lens information obtaining step, wherein inthe recording step, the moving image data is fragmented to record thefragments, and lens information obtained in the lens informationobtaining step is added to each fragment and recorded.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view showing the outer appearance of alens-interchangeable single-lens reflex digital camera;

FIG. 2 is a vertical sectional view showing the internal structure ofthe lens-interchangeable single-lens reflex digital camera;

FIG. 3 is a block diagram showing the circuit arrangement of thelens-interchangeable single-lens reflex digital camera;

FIG. 4 is a flowchart for explaining dust detection processing;

FIG. 5 is a view showing an example of the data format of dustcorrection data;

FIG. 6 is a flowchart for explaining details of a dust region obtainingroutine in step S27 of FIG. 4;

FIG. 7 is a view showing the processing unit of dust regiondetermination processing in step S62 of FIG. 6;

FIG. 8 is a view showing an outline of calculating the size of a dustregion in step S63 of FIG. 6;

FIG. 9 is a flowchart for explaining details of an image sensingprocessing routine in step S24 of FIG. 4;

FIG. 10 is a flowchart for explaining details of dust removalprocessing;

FIG. 11 is a flowchart for explaining details of an interpolationroutine;

FIG. 12 is a view for explaining the concept of metadata and media datain the MP4 file format or a similar file format;

FIG. 13 is a view for explaining the concept of Fragmented Movie;

FIG. 14 is a flowchart of basic processing in the first embodiment;

FIG. 15 is a view showing an example of the data format of dust positioncorrection data;

FIG. 16 is a chart showing moving image file fragmentation/generationprocessing in the first embodiment;

FIGS. 17A and 17B are a schematic view of a basic file structure in thefirst embodiment;

FIGS. 18A and 18B are a schematic view showing the second example of thefile structure in the first embodiment;

FIGS. 19A and 19B are a schematic view showing the third example of thefile structure in the first embodiment;

FIGS. 20A and 20B are a schematic view showing the fourth example of thefile structure in the first embodiment;

FIG. 21 is a block diagram showing the schematic system configuration ofan image processing apparatus;

FIG. 22 is a view showing an example of the GUI in the image processingapparatus;

FIG. 23 is a flowchart of basic processing in the second embodiment;

FIG. 24 is a flowchart showing a fragmentation method in zoom driving inthe third embodiment;

FIG. 25 is a flowchart showing a fragmentation method in zoom driving inthe fourth embodiment; and

FIG. 26 is a flowchart showing a fragmentation method in zoom driving inthe fifth embodiment.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will now be described in detailwith reference to the accompanying drawings.

(First Embodiment)

FIG. 1 is a perspective view showing the outer appearance of a digitalcamera 120 common to all the embodiments of the present invention. FIG.2 is a vertical sectional view of FIG. 1.

Referring to FIG. 1, the top of a camera body 100 includes an accessoryshoe 110, an optical viewfinder 104, an AE (Auto Exposure) lock button111, an AF distance measurement point selection button 113, and arelease button 114 for performing a shooting operation. The top of thecamera body 100 also includes an electronic dial 411, mode dial 60, andexternal display 409. The electronic dial 411 is a multifunctionalsignal input device for inputting a numerical value to the camera incombination with another operation button, or switching the shootingmode. The external display 409 is formed from a liquid crystal display,and displays shooting conditions (e.g., shutter speed, aperture value,and shooting mode), and other kinds of information.

The rear side of the camera body 100 includes an LCD monitor 417 fordisplaying a shot image and various setup windows, a playback switch 66for playing back a shot image on the LCD monitor 417, a singleshooting/continuous shooting switch 68, a four-way selector switch 116,a menu button 124, and a power switch 72.

The single shooting/continuous shooting switch 68 can set a singleshooting mode in which when the user presses a shutter switch SW2 64 (tobe described later), shooting of one frame is done and then the camerastands by, and a continuous shooting mode in which shooting continueswhile the user presses the shutter switch SW2 64.

The four-way selector switch 116 includes four buttons arranged on thetop, bottom, right, and left, and a SET button 117 arranged at thecenter. The user uses the four-way selector switch 116 to instruct thecamera to select or execute a menu item or the like displayed on the LCDmonitor 417.

The user uses the menu button 124 to display, on the LCD monitor 417, amenu window for making various settings of the camera. For example, whenselecting and setting the shooting mode, the user presses the menubutton 124, and operates the top, bottom, right, and left buttons of thefour-way selector switch 116 to select a mode he wants. While the modeis selected, the user presses the SET button 117, completing thesetting.

The LCD monitor 417 in the embodiment is of the transmission type. Byonly driving the LCD monitor, the user cannot see an image. The LCDmonitor 417 requires a backlight illumination unit 416 behind it, asshown in FIG. 2. The LCD monitor 417 and backlight illumination unit 416form an image display unit 28, as shown in FIG. 3.

As shown in FIG. 2, the image sensing apparatus according to theembodiment mainly includes the camera body 100 and alens-interchangeable lens unit 300. In FIG. 2, reference numeral 401denotes an imaging optical axis.

The lens unit 300 includes an imaging lens 310 formed from a pluralityof lenses, a stop 312, and a lens mount 306 which mechanically connectsthe lens unit 300 to the camera body 100. The lens unit 300 isdetachable from the camera body 100 via the lens mount 306.

In the camera body 100, a mirror 130 is inserted in the imaging opticalpath, and is movable between a position (position shown in FIG. 2, whichwill be called an inclined mirror position) where the mirror 130 guidesobject light traveling from the lens unit 300 to the optical viewfindersystem, and a position (to be called a retraction position) where itretracts from the imaging optical path. The mirror 130 may also be aquick return mirror or half-mirror.

Referring to FIG. 2, object light guided from the mirror 130 to theoptical viewfinder 104 forms an image on a focusing screen 204. Acondenser lens 205 improves the visibility of the viewfinder. Apentagonal roof prism 132 guides the object light having passed throughthe focusing screen 204 and condenser lens 205 to an eyepiece lens 208for viewfinder observation and the optical viewfinder 104.

A second curtain 209 and first curtain 210 form a shutter. The secondcurtain 209 and first curtain 210 are opened to expose, for a necessarytime, an image sensor 14 which is arranged behind them tophotoelectrically convert an object image. An optical low-pass filter418 is arranged in front of the image sensor 14, and adjusts the specialfrequency of the object image to be formed on the image sensor 14. Dust(foreign substance) which adversely affects a shot image is adhered tothe optical low-pass filter 418. Such dust appears as a shadow in anobject image formed on the image sensor 14, degrading the quality of theshot image.

A printed board 211 holds the image sensor 14. A display board 215,which is another printed board, is arranged behind the printed board211. The LCD monitor 417 and backlight illumination unit 416 arearranged on a surface of the display board 215 that is opposite to theprinted board 211.

A recording medium 200 records image data. The camera uses a cell(portable power supply) 86. The recording medium 200 and cell 86 aredetachable from the camera body.

FIG. 3 is a block diagram showing the circuit arrangement of thelens-interchangeable digital camera common to all the embodiments of thepresent invention.

The arrangement of the lens unit 300 will be explained.

The lens mount 306 incorporates various functions for electricallyconnecting the lens unit 300 to the camera body 100. In the lens mount306, an interface 320 connects the lens unit 300 to the camera body 100.A connector 322 electrically connects the lens unit 300 to the camerabody 100.

The connector 322 also has a function of exchanging control signals,status signals, and data signals between the camera body 100 and thelens unit 300 and receiving currents of various voltages. The connector322 may also communicate not only by telecommunication but also byoptical communication or speech communication.

A stop control unit 340 controls the stop 312 in cooperation with ashutter control unit 40 (to be described later) which controls a shutter12 of the camera body 100 based on photometry information from aphotometry control unit 46. A focus control unit 342 controls focusingof the imaging lens 310. A zoom control unit 344 controls zooming of theimaging lens 310.

A lens system control circuit 350 controls the overall lens unit 300.The lens system control circuit 350 has a memory for storing constants,variables, and programs for operations. The lens system control circuit350 also has a nonvolatile memory for holding identification information(e.g., a number unique to the lens unit 300), management information,functional information (e.g., a full-aperture value, minimum aperturevalue, and focal length), and current and past set values.

The arrangement of the camera body 100 will be described next.

A lens mount 106 mechanically connects the camera body 100 to the lensunit 300. The shutter 12 includes the second curtain 209 and firstcurtain 210. According to the single-lens reflex method, a light beamwhich has entered the imaging lens 310 is guided via the stop 312serving as a light quantity restriction unit, the lens mounts 306 and106, the mirror 130, and the shutter 12, and forms an optical image onthe image sensor 14.

An A/D converter 16 converts an analog signal output from the imagesensor 14 into a digital signal. A timing generator 18 supplies clocksignals and control signals to the image sensor 14, the A/D converter16, and a D/A converter 26. A memory control circuit 22 and systemcontrol circuit 50 control the timing generator 18.

An image processing circuit 20 executes predetermined pixelinterpolation processing and color conversion processing for data fromthe A/D converter 16 or data from the memory control circuit 22. Ifnecessary, the image processing circuit 20 performs predeterminedarithmetic processing using image data output from the A/D converter 16.Based on the obtained arithmetic result, the system control circuit 50can execute auto focus (AF) processing, auto exposure (AE) processing,and pre-electronic flash (EF) processing of TTL (Through The Lens)scheme to control the shutter control unit 40 and a focus adjusting unit42. The image processing unit 20 also executes predetermined arithmeticprocessing using image data output from the A/D converter 16, andperforms auto white balance (AWB) processing of TTL scheme based on theobtained arithmetic result.

In the example shown in FIG. 3 in the embodiment, the focus adjustingunit 42 and photometry control unit 46 are provided for exclusive use.Hence, AF processing, AE processing, and EF processing may also be doneusing not the image processing circuit 20 but the focus adjusting unit42 and photometry control unit 46. Alternatively, AF processing, AEprocessing, and EF processing may also be performed first using thefocus adjusting unit 42 and photometry control unit 46 and then usingthe image processing circuit 20.

The memory control circuit 22 controls the A/D converter 16, the timinggenerator 18, the image processing circuit 20, an image display memory24, the D/A converter 26, a memory 30, and a compression/decompressioncircuit 32. Image data output from the A/D converter 16 is written inthe image display memory 24 or memory 30 via the image processingcircuit 20 and memory control circuit 22 or via only the memory controlcircuit 22.

The image display unit 28 includes the LCD monitor 417 and backlightillumination unit 416. Display image data written in the image displaymemory 24 is displayed on the image display unit 28 via the D/Aconverter 26. The image display unit 28 sequentially displays sensedimage data, implementing an electronic viewfinder (EVF) function. Theimage display unit 28 can arbitrarily turn on/off its display inaccordance with an instruction from the system control circuit 50. Whendisplay is OFF, the power consumption of the camera body 100 can begreatly reduced.

The memory 30 stores shot still images and has a memory capacity enoughto store a predetermined number of still images. Even in continuousshooting or panoramic shooting for continuously shooting a plurality ofstill images, the memory 30 allows writing many images in it at highspeed. In moving image shooting, the memory 30 is used as a frame bufferfor continuously writing images at a predetermined rate. The memory 30is also available as the work area of the system control circuit 50.

The compression/decompression circuit 32 compresses/decompresses imagedata using a known compression method. The compression/decompressioncircuit 32 reads out an image from the memory 30, compresses ordecompresses it, and writes the processed data in the memory 30 again.

The shutter control unit 40 controls the shutter 12 in cooperation withthe stop control unit 340 which controls the stop 312 based onphotometry information from the photometry control unit 46. The focusadjusting unit 42 executes AF (Auto Focus) processing. According to thesingle-lens reflex method, a light beam which has entered the imaginglens 310 of the lens unit 300 is guided via the stop 312, the lensmounts 306 and 106, the mirror 130, and a focus adjusting sub-mirror(not shown). The focus adjusting unit 42 detects the focus state of animage formed by the light beam as an optical image.

The photometry control unit 46 executes AE (Auto Exposure) processing.According to the single-lens reflex method, a light beam which hasentered the imaging lens 310 of the lens unit 300 is guided via the stop312, the lens mounts 306 and 106, the mirror 130, and a photometrysub-mirror (not shown). The photometry control unit 46 measures theexposure state of an image formed by the light beam as an optical image.An electronic flash 48 has an AF auxiliary light projecting function andan electronic flash control function. The photometry control unit 46also has an EF (Electronic Flash control) processing function incooperation with the electronic flash 48.

AF control may also be done using the measurement result of the focusadjusting unit 42 and an arithmetic result obtained by arithmeticallyprocessing image data from the A/D converter 16 by the image processingcircuit 20. Exposure control may also be performed using the measurementresult of the photometry control unit 46 and an arithmetic resultobtained by arithmetically processing image data from the A/D converter16 by the image processing circuit 20.

The system control circuit 50 controls the overall camera body 100 andincorporates a known CPU. A memory 52 stores constants, variables, andprograms for the operation of the system control circuit 50.

A notification unit 54 notifies the outside of operation states andmessages using a text, image, and sound in accordance with execution ofa program by the system control circuit 50. The notification unit 54 is,e.g., a display unit such as an LCD or LED for providing a visualdisplay, or a sound generation element for generating a notification bysound. The notification unit 54 includes one or a combination of them.Especially when the notification unit 54 is a display unit, it isarranged at one or a plurality of positions near an operation unit 70 ofthe camera body 100, like the external display 409, where the user caneasily see a notification. Some functions of the notification unit 54are arranged in the optical viewfinder 104.

Among the display contents of the notification unit 54, the imagedisplay unit 28 such as an LCD presents a display associated withshooting modes including single shooting/continuous shooting and selftimer. The image display unit 28 also presents a display associated withrecording including the compression ratio, the number of recordingpixels, the number of recorded images, and the number of recordableimages. Further, the image display unit 28 presents a display associatedwith shooting conditions including the shutter speed, aperture value,exposure compensation, brightness correction, external flash lightemission amount, and red eye mitigation. The image display unit 28 alsodisplays macro shooting, buzzer setting, battery level, error message,information by a plurality of digits, and the attached/detached statesof the recording medium 200 and a PC 210. In addition, the image displayunit 28 displays the attached/detached state of the lens unit 300,communication I/F operation, date and time, and the connection state ofan external computer.

Some of the display contents of the notification unit 54 are indicatedin the optical viewfinder 104, which include in-focus, ready forshooting, camera shake warning, flash charge, flash charge completion,shutter speed, aperture value, exposure compensation, and recordingmedium write operation.

A nonvolatile memory 56 is an electrically erasable/programmable memorysuch as an EEPROM, and stores programs (to be described later) and thelike.

Reference numerals 60, 62, 64, 66, 68, and 70 denote operation units forinputting various kinds of operation instructions to the system controlcircuit 50. Each operation unit includes one or a combination of aswitch, dial, touch panel, pointing by line-of-sight detection, andvoice recognition device.

These operation units will be described here in detail.

The mode dial switch 60 can selectively set a shooting mode such as anautomatic shooting mode, programmed shooting mode, shutter speedpriority shooting mode, stop priority shooting mode, manual shootingmode, or focal depth priority (depth) shooting mode. The mode dialswitch 60 can also selectively set a shooting mode such as a portraitshooting mode, landscape shooting mode, closeup shooting mode, sportsshooting mode, nightscape shooting mode, panoramic shooting mode, andmoving image shooting mode.

The shutter switch SW1 62 is turned on by operating the release button114 halfway (e.g., half stroke) to designate the start of an operationsuch as AF processing, AE processing, AWB processing, or EF processing.

The shutter switch SW2 64 is turned on by operating the release button114 completely (e.g., full stroke) to designate the start of a series ofprocesses including exposure, development, and recording. In theexposure processing, a signal read out from the image sensor 14 iswritten in the memory 30 via the A/D converter 16 and memory controlcircuit 22. Then, the development processing is done using calculationby the image processing circuit 20 or memory control circuit 22. In therecording processing, image data is read out from the memory 30,compressed by the compression/decompression circuit 32, and written inor transmitted to the recording medium 200 or PC 210.

The playback switch 66 designates the start of a playback operation ofreading out an image shot in a shooting mode from the memory 30,recording medium 200, or PC 210 and displaying it on the image displayunit 28. The playback switch 66 can set a functional mode such as aplayback mode, multiwindow playback/erase mode, or PC-connected mode.

The single shooting/continuous shooting switch 68 can set a singleshooting mode in which when the user presses the shutter switch SW2 64,shooting of one frame is done, and then the camera stands by, and acontinuous shooting mode in which shooting continues while the userpresses the shutter switch SW2 64.

The operation unit 70 includes various buttons and a touch panel. Forexample, the operation unit 70 includes a live view start/stop button, amovie recording start/stop button, the menu button 124, the SET button117, a multiwindow playback/page feed button, an electronic flashsetting button, a single shooting/continuous shooting/self timer switchbutton, the four-way selector switch 116, the AE (Auto Exposure) lockbutton 111, the AF distance measurement point selection button 113, andthe electronic dial 411. Further, the operation unit 70 includes aplayback image move + (plus) button, playback image move − (minus)button, shooting image quality selection button, exposure compensationbutton, brightness correction button, external flash light emissionamount setting button, and date/time setting button. When a rotary dialswitch is used for the top, bottom, right, and left buttons of thefour-way selector switch 116, it allows the user to more easily selectnumerical values and functions.

In addition, the operation unit 70 includes an image display ON/OFFswitch for turning on/off the image display unit 28, and a quick reviewON/OFF switch for setting a quick review function of automaticallyplaying back shot image data immediately after shooting. The operationunit 70 also includes a compression mode switch for selecting acompression ratio for JPEG compression, or a RAW mode to directlydigitize a signal from the image sensor and record it on a recordingmedium. Further, the operation unit 70 includes an AF mode settingswitch capable of setting a one-shot AF mode or a servo AF mode. In theone-shot AF mode, the auto focus operation starts when the user pressesthe shutter switch SW1 62. Once an in-focus state is obtained, thisstate is kept held. In the servo AF mode, the auto focus operationcontinues while the user presses the shutter switch SW1 62. Theoperation unit 70 also includes a setting switch capable of setting adust information obtainment mode to sense a dust detection image andobtain dust information, as will be described later.

The power switch 72 can selectively set the power ON or power OFF modeof the camera body 100. The power switch 72 can also selectively set thepower ON or power OFF mode of each of various accessories including thelens unit 300, an external flash 112, the recording medium 200, and thePC 210 which are connected to the camera body 100.

A power supply control unit 80 includes a cell detection circuit, DC/DCconverter, and switching circuit for switching a block to be energized.The power supply control unit 80 detects attachment/detachment of acell, the type of cell, and the battery level. The power supply controlunit 80 controls the DC/DC converter based on the detection result andan instruction from the system control circuit 50. The power supplycontrol unit 80 supplies a necessary voltage to the respective unitsincluding a recording medium for a necessary period.

Reference numerals 82 and 84 denote connectors; and 86, a power supplyunit formed from a primary cell (e.g., alkaline cell or lithium cell), asecondary cell (e.g., an NiCd cell, NiMH cell, Li-ion cell, orLi-polymer cell), or an AC adapter.

Reference numerals 90 and 94 denote interfaces with a PC and a recordingmedium such as a memory card or hard disk; and 92 and 96, connectors toconnect a PC and a recording medium such as a memory card or hard disk.A recording medium attachment detection circuit 98 detects whether therecording medium 200 and/or PC 210 is connected to the connectors 92and/or 96.

In the embodiment, the camera has two interfaces and two connectors toconnect a recording medium. However, the numbers of interfaces andconnectors to connect a recording medium are arbitrary, and the cameracan have one or a plurality of interfaces or connectors. Interfaces andconnectors of different standards may also be combined.

Interfaces and connectors complying with various storage mediumstandards are available. Examples are a PCMCIA (Personal Computer MemoryCard International Association) card, CF (Compact Flash®) card, and SDcard. When the interfaces 90 and 94 and the connectors 92 and 96 complywith the standard of the PCMCIA card or CF® card, they can connectvarious kinds of communication cards. Examples of the communicationcards are a LAN card, modem card, USB (Universal Serial Bus) card, andIEEE (Institute of Electrical and Electronic Engineers) 1394 card. AP1284 card, SCSI (Small Computer System Interface) card, and PHS arealso usable. Various kinds of communication cards can be connected totransfer image data and management information associated with it toanother computer or a peripheral device such as a printer.

The optical viewfinder 104 can display an optical image formed by alight beam which has entered the imaging lens 310 and is guided via thestop 312, lens mounts 306 and 106, and mirrors 130 and 132 by thesingle-lens reflex method. Only with the optical viewfinder, the usercan take a picture without using the electronic viewfinder function ofthe image display unit 28. The optical viewfinder 104 displays somefunctions of the notification unit 54 such as the in-focus state, camerashake warning, flash charge, shutter speed, aperture value, and exposurecompensation.

The external flash 112 is attached via the accessory shoe 110.

An interface 121 connects the camera body 100 to the lens unit 300 inthe lens mount 106.

A connector 122 electrically connects the camera body 100 to the lensunit 300. A lens attachment detection unit (not shown) detects whetherthe lens unit 300 is attached to the lens mount 106 and connector 122.The connector 122 also has a function of transmitting control signals,status signals, data signals, and the like between the camera body 100and the lens unit 300 and also supplying currents of various voltages.

The memory 30 of the camera body 100 stores various kinds of opticalinformation (e.g., aperture value, zoom position, pupil distance, andfocal length) of the lens unit 300 that are communicated via theconnector 122. In some cases, the camera requests communication of theinformation. Every time the information is updated, the lens maycommunicate it.

The connector 122 may also communicate not only by telecommunication butalso by optical communication or speech communication.

The recording medium 200 is, e.g., a memory card or hard disk. Therecording medium 200 includes a recording unit 202 formed from asemiconductor memory or magnetic disk, an interface 204 with the camerabody 100, and a connector 206 to connect the camera body 100.

The recording medium 200 can be a memory card (e.g., PCMCIA card orCompact Flash®), or a hard disk. The recording medium 200 may also be amicro DAT, a magnetooptical disk, an optical disk (e.g., CD-R or CD-RW),or a phase-change optical disk (e.g., DVD).

The PC 210 includes a recording unit 212 formed from a magnetic disk(HD), an interface 214 with the camera body 100, and a connector 216 toconnect the camera body 100. The interface 214 can be a USB orinterface, but is not particularly limited.

Processing of performing image processing to eliminate the influence ofdust on an optical member such as a low-pass filter or cover glassarranged in front of the image sensor in the image sensing apparatushaving the above-described arrangement will be described next.

In the embodiment, the camera shoots a dust detection image (foreignsubstance detection image) for obtaining dust information (foreignsubstance information) serving as information on the adhesion positionand size of dust (foreign substance). Then, the dust detection image isextracted to generate dust data. The dust detection image is preferablyobtained by shooting a surface as uniformly bright as possible. However,the uniformity need not be strict because it is desirable to easilyshoot the image in a familiar place. For example, the embodiment assumesshooting a blue sky or white wall.

An example of processing to detect the position of dust adhered in theimage sensing optical system will be explained with reference to theflowchart of FIG. 4. The system control circuit 50 performs thisprocessing by executing a dust detection processing program stored inthe nonvolatile memory 56.

In the dust detection processing, a dust detection image is shot. Whenperforming the dust detection processing, the user prepares for dustdetection by setting the camera to direct the imaging optical axis 401of the lens unit 300 to the exit surface of a surface light source or asurface with a uniform color, like a white wall. The user also preparesfor dust detection by attaching a dust detection light unit (compactpoint light source attached instead of the lens) to the lens mount 106.The light source of the light unit is, e.g., a white LED, and the sizeof the light emitting surface is desirably adjusted to comply with apredetermined aperture value (e.g., F32).

The embodiment will explain dust detection using a general imaging lens.The dust detection may also be done by attaching the light unit to thelens mount 106. In the embodiment, a dust detection image is an imagewith a uniform color.

After the end of preparation, when the user instructs the camera via thefour-way selector switch 116 to start dust detection processing, thesystem control circuit 50 sets the stop first. The imaging state of dustnear the image sensor changes depending on the aperture value of thelens, and its position changes depending on the lens pupil position. Forthis reason, dust correction data needs to hold an aperture value andlens pupil position in detection, in addition to the position and sizeof dust.

However, dust correction data need not always hold an aperture value ifit is set to always use the same aperture value even for differentlenses when creating dust correction data. As for the pupil position,dust correction data need not always hold it if the light unit is usedor the use of only a specific lens is permitted.

In other words, if the use of a plurality of types of lenses ispermitted or the aperture value is properly changed when creating dustcorrection data, the dust correction data needs to hold an aperturevalue and lens pupil position in detection. Note that the pupil positionmeans a distance from the image sensing plane (focal plane) of the exitpupil.

For example, F32 is designated (step S21).

Then, the system control circuit 50 causes the stop control unit 340 viathe connector 122 to control the aperture blades of the lens unit 300and set the stop to the aperture value designated in step S21 (stepS22). The system control circuit 50 causes the focus control unit 342 toset the focus position to infinity (step S23).

After setting the aperture value and focus position of the imaging lens,the system control circuit 50 executes shooting in the dust detectionmode (step S24). Details of the image sensing processing routine in stepS24 will be explained with reference to FIG. 9. The memory 30 stores theshot image data.

After the end of shooting, the system control circuit 50 obtains anaperture value and lens pupil position in shooting (step S25). Thesystem control circuit 50 reads out, to the image processing circuit 20,data corresponding to each pixel of the shot image stored in the memory30 (step S26). The image processing circuit 20 performs processing shownin FIG. 6, obtaining the position and size of a pixel where dust exists(step S27). The nonvolatile memory 56 registers the position and size ofthe pixel where dust exists, which have been obtained in step S27, andthe aperture value and lens pupil position information which have beenobtained in step S25 (step S28). When the foregoing light unit is used,no lens information can be obtained. When no lens information can beobtained, the system control circuit 50 determines that the light unithas been used. Then, the nonvolatile memory 56 registers predeterminedlens pupil position information, and an aperture value calculated fromthe light source diameter of the light unit.

In step S28, the system control circuit 50 compares the position of adefective pixel (pixel defect) in the manufacture that is recorded inadvance in the nonvolatile memory 56 with the position of the readoutpixel data, and determines whether the target pixel is defective. Thenonvolatile memory 56 may also register only the position of a regiondetermined not to have a pixel defect.

FIG. 5 shows an example of the data format of dust correction datastored in the nonvolatile memory 56. As shown in FIG. 5, the dustcorrection data stores lens information, dust position, and sizeinformation obtained when a detection image was shot.

More specifically, an actual aperture value (F-number) used to shoot adetection image, and lens pupil position at that time are stored as lensinformation obtained when a detection image was shot. Then, the number(integer value) of detected dust regions is stored in the storage area.Subsequently, concrete parameters of each dust region are repetitivelystored by the number of dust regions. The parameters of each dust regionare a set of three numerical values: the radius (e.g., 2 bytes) of dust,the x-coordinate (e.g., 2 bytes) of the center in the effective imageregion, and the y-coordinate (e.g., 2 bytes) of the center.

If the dust correction data size is limited by the capacity of thenonvolatile memory 56 or the like, data are stored preferentially fromthe start of dust regions obtained in step S27. This is because dustregions are sorted in order from the most conspicuous dust in the dustregion obtaining routine of step S27, which will be described later.

Details of the dust region obtaining routine in step S27 of FIG. 4 willbe explained with reference to FIGS. 6 to 8.

As shown in FIG. 7, readout image data is rasterized in the memory 30,and processed for each predetermined block in order to cope with limbdarkening arising from the lens or sensor characteristic. Limb darkeningis a phenomenon in which the luminance at the periphery of the lensbecomes lower than that at the center. It is known that limb darkeningcan be reduced to a certain degree by setting the lens to a largeaperture value. However, even if the lens is set to a large aperturevalue, dust at the periphery may not be accurately detected depending onthe lens when the position of dust in a shot image is determined basedon a predetermined threshold value. From this, the influence of limbdarkening is reduced by dividing an image into blocks.

If an image is simply divided into blocks, the dust detection result maychange between blocks when the threshold value changes between them. Toprevent this, blocks are made to overlap each other. A pixel determinedto have dust in either block of the overlapping region is handled as adust region.

Determination of a dust region in a block is executed according to theprocessing sequence shown in FIG. 6. A maximum luminance Lmax andaverage luminance Lave in each block are calculated. A threshold valueT1 in each block is calculated byT1=Lave×0.6+Lmax×0.4

A pixel whose luminance does not exceed the threshold value isdetermined as a dust pixel (step S61). Each isolated region formed fromdust pixels is defined as a dust region di (i=0, 1, . . . , n) (stepS62). As shown in FIG. 8, a maximum value Xmax and minimum value Xmin ofthe horizontal coordinates of pixels falling within a dust region, and amaximum value Ymax and minimum value Ymin of their vertical coordinatesare obtained. A radius ri representing the size of the dust region di iscalculated (step S63):ri=√[{(Xmax−Xmin)/2}²+{(Ymax−Ymin)/2}²]

FIG. 8 shows the relationship between Xmax, Xmin, Ymax, Ymin, and ri.

In step S64, the average luminance value of each dust region iscalculated.

The size of dust correction data is sometimes limited by the capacity ofthe nonvolatile memory 56 or the like. To cope with this case, pieces ofdust position information are sorted by the size or average luminancevalue of the dust region (step S65). In the embodiment, pieces of dustposition information are sorted in descending order of ri. If all dustregions have the same ri, they are sorted in ascending order of theaverage luminance value. As a result, noticeable dust can bepreferentially registered in dust correction data. Di represents asorted dust region, and Ri represents the radius of the dust region Di.

If there is a dust region larger than a predetermined size, it may alsobe excluded from the sorting targets and added to the end of thesorted-dust region list. A large dust region may degrade the imagequality if it undergoes interpolation processing later. It is desirableto correct such a large dust region finally.

Details of the image sensing processing routine in step S24 of FIG. 4will be explained with reference to the flowchart shown in FIG. 9. Thesystem control circuit 50 performs this processing by executing an imagesensing processing program stored in the nonvolatile memory 56.

When the image sensing processing routines starts, the system controlcircuit 50 operates the mirror 130 shown in FIG. 3 to flip it up andretract it from the imaging optical path in step S201.

In step S202, the image sensor 14 starts accumulating charges. In stepS203, the shutter 12 shown in FIG. 3 travels to perform exposure. Instep S204, the charge accumulation of the image sensor 14 ends. In stepS205, an image signal is read out from the image sensor 14, and imagedata processed by the A/D converter 16 and image processing circuit 20is temporarily stored in the memory 30.

In step S206, read of all image signals from the image sensor ends. Instep S207, the mirror 130 flips down and returns to the inclined mirrorposition. Then, a series of image sensing operations ends.

In step S208, the system control circuit 50 determines whether theshooting mode is still image shooting or dust detection image shooting.If the shooting mode is still image shooting, the process advances tostep S209 to record the shot still image on the recording medium 200.

The first embodiment is directed to a method of performing imageprocessing to correct an image quality degraded by dust when shooting amoving image. Prior to a description of moving image processing, stillimage processing will be explained.

The sequence of an operation to perform dust removal for a still imagefile by image processing using the above-mentioned dust correction datawill be explained with reference to FIG. 10.

A still image file to undergo dust removal processing is designated andloaded into an apparatus (which may be the image processing circuit 20in the camera or an image processing apparatus outside the camera) forperforming dust removal processing (step S1801).

The apparatus for performing dust removal processing obtains dustcorrection data created in step S65 of FIG. 6 (step S1802).

A coordinate sequence Di (i=1, 2, . . . , n), a radius sequence Ri (i=1,2, . . . , n), an aperture value f1, and a lens pupil position L1 areobtained from the dust correction data obtained in step S1802 (stepS1803). Ri represents the size of dust at the coordinates Di calculatedinstep S65 of FIG. 6. In step S1804, an aperture value f2 and lens pupilposition L2 in shooting are obtained. In step S1805, Di is converted bythe following equation. Converted coordinates Di′ and a converted radiusRi′ are defined byDi′(x,y)=(L2×(L1−H)×d/((L2−H)×L1))×Di(x,y)Ri′=(Ri×f1/f2+3)  (1)where d is the distance from the image center to the coordinates Di, andH is the distance from the surface of the image sensor 14 to dust.

The unit is a pixel, and “+3” for Ri′ means a margin.

In step S1806, dust in a region defined by the coordinates Di′ andradius Ri′ is detected, and if necessary, interpolation processing isapplied. Details of the interpolation processing will be describedlater. In step S1807, it is determined whether all coordinates haveundergone the dust removal processing. If it is determined that allcoordinates have been processed, the process ends. If it is determinedthat all coordinates have not been processed, the process returns tostep S1806.

Details of the dust region interpolation processing will be explained.FIG. 11 is a flowchart showing the sequence of the interpolationroutine.

In step S1901, determination of the dust region is done. The dust regionis a region which satisfies all the following conditions:

(1) a region whose luminance is lower than a threshold value T2calculated using an average luminance Yave and maximum luminance Ymax ofpixels falling within a region defined by the center coordinates Di′ andradius Ri′ (Di′ and Ri′ calculated by equation (1)) calculated in stepS1805 of FIG. 10:T2=Yave×0.6+Ymax×0.4

(2) a region which does not contact a circle defined by the centercoordinates Di′ and radius Ri′.

(3) a region whose radius value calculated by the same method as stepS63 in FIG. 6 is equal to or larger than x1 pixels and smaller than x2pixels in an isolated region which is selected based on condition (1)and formed from low-luminance pixels.

(4) a region containing the center coordinates Di of the circle.

In the first embodiment, x1 represents three pixels, and x2 represents30 pixels. With this setting, only a small isolated region can behandled as a dust region. When no lens pupil position can be accuratelyobtained, condition (4) may also be eased. For example, when the regionof interest contains the coordinates of a range of ±3 pixels from thecoordinates Di in both the X and Y directions, it is determined as adust region.

If such a region exists in step S1902, the process advances to stepS1903 to perform dust region interpolation. If no such region exists,the process ends. The dust region interpolation processing executed instep S1903 adopts a known defective region interpolation method. Anexample of the known defective region interpolation method is patternreplacement disclosed in Japanese Patent Laid-Open No. 2001-223894. InJapanese Patent Laid-Open No. 2001-223894, a defective region isspecified using infrared light. In the embodiment, a dust regiondetected in step S1901 is handled as a defective region, andinterpolated by pattern replacement using normal surrounding pixels. Fora pixel which cannot be interpolated by pattern replacement, p normalpixels are selected sequentially from one closest to the pixel to beinterpolated in image data having undergone pattern correction, and thetarget pixel is interpolated using the average color of them.

Next, MP4 will be explained. The MP4 is a moving image file format usedto record moving image data in recent digital cameras, digital videocameras, and the like.

The MP4 file format (see ISO/IEC 14496-14; “Informationtechnology—Cording of audio-visual Objects—Part 14: MP4 file format”;ISO/IEC; 2003-11-24) is extended from a general-purpose file format “ISOBase Media File Format” (see ISO/IEC 14496-12; “Informationtechnology—Cording of audio-visual Objects—Part 12: ISO base media fileformat”; ISO/IEC; 2004-01-23). The MP4 file format aims at recordingfiles of moving image/audio contents data such as MPEG data standardizedby ISO/IEC JTC1/SC29/WG11 (International Organization forStandardization/International Engineering Consortium). The firstembodiment is applicable not only to MP4 but also to another similarfile format. For example, ISO has established standards “Motion JPEG2000 file format” (ISO/IEC 15444-3) and “AVC file format” (ISO/IEC14496-15) as file format standards having the same basic structure asthat of MP4.

FIG. 12 is a conceptual view for explaining the data structure of theMP4 file format.

An MP4 file 1001 contains metadata (header information) 1002representing the physical position, temporal position, andcharacteristic information of video and audio data, and media data 1003representing the entities of encoded video and audio data. In the MP4format, a presentation of whole contents is called a “movie”, and thatof a media stream which forms the contents is called a “track”. Themetadata 1002 typically contains a video track 1004 for logicallyhandling entire moving image data, and an audio track 1005 for logicallyhandling entire audio data. The video track 1004 and audio track 1005have almost the same configuration contents. More specifically,respective tracks record various kinds of metadata information of actualmedia data. The contents are slightly different in accordance with thecharacteristic of media data.

Data contained in the video track 1004 include, for example,configuration information of a so-called decoder for decoding encodeddata, and information on the rectangular size of a moving image. Inaddition, the data include an offset 1006 representing a position in afile where media data is actually recorded, and a sample size 1007representing the size of each frame data (also called a picture) ofmedia data. The video track 1004 also records a time stamp 1008representing the decoding time of each frame data.

The media data 1003 records the entities of moving image data and audiodata in a data structure “chunk” which successively records one or more“samples” representing the basic unit of encoded data. The chunkincludes a video chunk 1009 containing media data of a moving image, andan audio chunk 1010 containing media data of audio data in accordancewith the track of the metadata 1002.

In the structure shown in FIG. 12, the video chunk 1009 and audio chunk1010 are alternately recorded (interleaved), but the recording positionsand order are not limited to those shown in FIG. 12. The recordingpositions and order shown in FIG. 12 are merely an example of a generalrecording format. However, this interleave arrangement can improve theaccessibility of data recorded in a file because moving image data andaudio data to be played back almost simultaneously are arranged at closepositions. Thus, the interleave arrangement is very popular.

The chunk contains one or more samples of each media data. For example,as shown in FIG. 12, the video chunk 1009 successively records videosamples (frames) 1011. In general, each video sample (frame) 1011corresponds to one frame data (picture) of video data. Each track andeach chunk are associated as follows.

For example, for moving image data, information contained in the videotrack 1004 includes information on each video chunk 1009 contained inthe media data 1003. The offset 1006 is formed from a table ofinformation representing the relative position of the video chunk 1009in a corresponding file. By looking up each entry of the table, theposition of an actual video chunk can be specified regardless of wherethe video chunk is recorded. The sample size 1007 describes, in a table,the sizes of respective samples, i.e., video frames contained in aplurality of chunks. The video track 1004 also describes information onthe number of samples contained in each chunk. From this information,samples contained in each video chunk 1009 can be obtained accurately.The time stamp 1008 records the decoding time of each sample in a tableas the difference between samples. By looking up the table, a so-calledtime stamp of each sample can be obtained by calculating the accumulatedtime. The relationship between the track and the chunk is defined sothat it is also similarly established between even the audio track 1005and the audio chunk 1010. In the MP4 file format and ISO Base Media FileFormat, the metadata 1002 and media data 1003 can provide encoded datain a necessary unit from an arbitrary position together with additionalinformation such as the time stamp. For descriptive convenience, not allpieces of standardized recording information have been described.Details of the definition contents of the standard can be acquired froma corresponding section of ISO/IEC 14496.

In the MP4 file format, data recorded in a file is described in a datastructure “BOX”. Data of each BOX is recorded in a file. The BOX isformed from the following fields:

-   -   Size: the size of the entire BOX including the size field        itself.    -   Type: a 4-byte type identifier representing the type of BOX. In        general, the type identifier is made up of four alphanumeric        characters.

Other fields are options depending on BOX, so a description thereof willbe omitted.

Data recorded in a file is held in a different type of BOX in accordancewith the type of data. For example, the media data 1003 is recorded asMedia Data BOX (the type field=‘mdat’: When an identifier representing aBOX type is used in the following description, it expresses a BOX ofthis type). The metadata 1002 is recorded as a movie BOX ‘moov’ whichstores metadata information of whole contents. Information on theabove-described chunk and sample is also recorded as BOX having a uniqueidentifier in moov for each track.

The MP4 file format not only records all metadata in moov, but alsopermits dividing metadata into a plurality of areas in time series andrecording them. This format is called “Fragmented Movie”.

FIG. 13 shows the file structure of the fragmented movie format. Thefragmented movie format allows dividing media data and metadata ofcontents by an arbitrary time. “Fragments” are recorded from the startof a file in time series. For example, in FIG. 13, moov 1101 representsmetadata of the first fragment, and holds information on data containedin mdat 1102. moof 1103 subsequent to the mdat 1102 represents metadataof the second fragment, and holds information on mdat 1104. In thisways, fragments are recorded. When the fragmented movie format isemployed, movie extends Box (‘mvex’) 1105 representing the presence of afragment needs to be added to the moov 1101. Information contained inthe mvex 1105 is, e.g., the duration (time length) of whole contentsincluding all fragments. In a file of the MP4 file format, a variety ofattributes associated with media data are held as a metadata areaseparately from the media data. Thus, desired sample data can be easilyaccessed regardless of how to physically store media data.

In the following description, the moving image file format used torecord moving image data and audio data in the first embodiment is theMP4 fragmented movie format as shown in FIG. 13. A method of associatingthe above-described dust correction data with the video sample (frame)1011 in moving image recording will be explained.

The method according to the first embodiment is also applicable tostandards which adopt file formats and architectures similar to thosedefined in MP4, such as the standards “Motion JPEG 2000 file format”(ISO/IEC 15444-3) and “AVC file format” (ISO/IEC 14496-15), and a 3GPP(3rd Generation Partnership Project) file format serving as a movingimage file which is constrained on the premise of the use on wirelessterminals including third-generation cell phones (see 3GPP TS 26.244“Technical Specification Group Services and System Aspects Transparentend-to-end packet switched streaming service (PSS); 3GPP file format(3GP) (Release 6)” 3rd Generation Partnership Project; 2003-02-28).

FIG. 14 is a flowchart showing processing to associate dust correctiondata with the frame 1011 and record a moving image. The system controlcircuit 50 performs this processing by executing a moving imagerecording processing program stored in the nonvolatile memory 56. Assumethat the nonvolatile memory 56 stores dust correction data. Also assumethat the memory 30 has already stored an aperture value (F-number) andlens pupil position as lens information of a lens attached at the startof moving image shooting. The lens information is copied to the memory52 at the start of moving image recording. The system control circuit 50obtains the lens information by communicating with the lens unit 300.

To shoot a moving image, the user needs to change the shooting mode froma still image shooting mode to a moving image shooting mode using themenu button 124 or mode dial 60. When the moving image shooting mode isset, the system control circuit 50 flips up the mirror 130 to retract itfrom the imaging optical path. The system control circuit 50 opens theshutter 12 to expose the image sensor 14 to object light. Image dataobtained by exposure are successively written at a predetermined rate inthe memory 30 serving as a frame buffer. The LCD monitor 417 functionsas an electronic viewfinder (EVF) to sequentially display the writtenimage data. In the moving image shooting mode, the operation unit 70detects whether the user has pressed the moving image recording startbutton (e.g., he has pressed the SET button 117 in the moving imageshooting mode). If so, moving image shooting starts to sequentiallyrecord image data on the recording medium 200 in the MP4 file format.

Referring back to FIG. 14, when moving image shooting starts uponpressing the moving image recording button in the moving image shootingmode, a new file is generated first. moov serving as the BOX of metadataof the first fragment and mdat serving as the BOX of media data arecreated (step S1201). Then, dust position correction data is created(step S1202). FIG. 15 shows an example of the data format of the createddust position correction data.

As shown in FIG. 15, the dust position correction data stores anaperture value and lens pupil position information serving as lensinformation of a lens used in moving image shooting, and the dustcorrection data shown in FIG. 5. The memory 52 stores the created dustposition correction data.

In step S1203, the dust position correction data stored in the memory 52is read and written in the moov of metadata of the current fragment,like dust position correction data 1502 in FIG. 17A. In this case, thesystem control circuit 50 functions as an information recording unit andfragment information storage unit. Note that the data structure in FIGS.17A and 17B will be described later.

After moving image shooting, image processing, and compressionprocessing (step S1204), moving image data is written in mdat of thecurrent fragment (step S1205). In this case, the system control circuit50 functions as a fragment recording unit.

Then, it is determined whether the user has requested the end of movingimage recording, i.e., he has pressed a moving image recording stopbutton (e.g., he has pressed the SET button 117 during moving imagerecording) (step S1206). If the user has requested the end of movingimage recording, the process ends (step S1210). If the user has notrequested the end, it is checked whether lens information has beenupdated (step S1207). The lens information is updated upon a change ofthe lens pupil position when the user operates the lens to zoom in/outthe object image, or a change of the aperture value by the user with anoperation member such as the electronic dial 411. The zoom control unit344 notifies the system control circuit 50 of a change of the pupilposition via the connectors 322 and 122. Also, the system controlcircuit 50 is notified of a change of the aperture value as signals ofmany switches including the electronic dial 411. In this case, the zoomcontrol unit 344 and operation unit 70 function as a lens informationupdate notification unit.

Upon receiving the notification, the system control circuit 50 functionsas a lens information obtaining unit. The system control circuit 50stores the notified current lens information in the memory 30, andrewrites it over lens information stored in the memory 52. When thephotometry control unit 46 detects an abrupt change of the brightness ofan object, it notifies the system control circuit 50 of this. Then, thesystem control circuit 50 causes the stop control unit 340 to drive andcontrol the aperture blades. The system control circuit 50 obtainsnotified lens information. When the lens pupil position changes uponauto focus (AF) driving of the lens, driving of an anti-vibration shiftlens for preventing camera shake, or the like, the zoom control unit 344and focus control unit 342 notify the system control circuit 50 of this,and the system control circuit 50 obtains lens information.

If update of the lens information is detected, the lens information tobe stored in the memory 52 overwrites an aperture value and lens pupilposition used when the dust position correction data was obtained (FIG.15) (step S1208). moof serving as the BOX of metadata of a new fragmentand mdat serving as the BOX of media data are added to the currentfragment during the write, updating the fragment in which the writeposition is created (step S1209). In this case, the system controlcircuit 50 functions as a fragment creation unit and fragment changecontrol unit. Thereafter, the process returns to step S1203 to write thedust position correction data updated in step S1208 in moof of metadataof the added fragment, like dust position correction data 1503 in FIG.17A.

If no update of lens information is detected in step S1207, moving imageshooting, image processing, and compression processing are performed(step S1204) without fragmentation. Moving image data is written in mdatof the current fragment (step S1205).

The series of processes (steps S1203, S1204, S1205, S1206, S1207, S1208,and S1209) is repeated until the user issues an end request.

Although not described in detail, a moving image file created uponreceiving an end request (step S1210) records various kinds of metadatainformation in moov and moof of respective fragments, mvex necessary forthe fragment format, and media data in mdat so as to be compatible withthe standard.

In this example, dust position correction data is recorded for eachfragment. However, dust correction data does not change during movingimage shooting. For this reason, dust position correction data may alsobe recorded in only moov of the first metadata in the format of FIG. 15.In this case, only an aperture value and lens pupil position duringshooting, which change during moving image shooting, are recorded inmoof of metadata after fragmentation.

FIG. 16 is a chart showing an example of fragmentation of a generatedmoving image file. Recording starts at time 1301 and stops at time 1304.Fragmentation events upon detecting changes of lens information such aszoom-in/out and a change of the aperture value occur at time 1302 andtime 1303. A first fragment 1305 stores dust position correction dataincluding lens information, and moving image data from the recordingstart time 1301 to the time 1302 when the first fragmentation eventoccurs.

When the first fragmentation event occurs (the time 1302), a secondfragment 1306 is generated as a new fragment. The second fragment 1306stores dust position correction data including lens information, andmoving image data from the first fragmentation event generation time1302 to the second fragmentation event generation time 1303.

When the second fragmentation event occurs (the time 1303), a thirdfragment 1307 is generated as a new fragment. The third fragment 1307stores dust position correction data including lens information, andmoving image data from the second fragmentation event generation time1303 to the time 1304 when the user requests the stop of recording.

In this way, one moving image file with a plurality of fragments such asthe first fragment 1305, second fragment 1306, and third fragment 1307generated every time a change of lens information is detected iscreated.

Instead of one moving image file, not new fragments but new moving imagefiles may also be created at the timings of fragmentation events (thetime 1302 and time 1303). When creating a new moving image file, a newfile is created in step S1209. In step S1203, dust position correctiondata is always added to moov of metadata. In step S1210, a plurality ofmoving image files are generated.

FIGS. 17A and 17B are a schematic view for explaining the data structureof the MP4 file format in the first embodiment. FIGS. 17A and 17B are aschematic view when two fragmentation events (the time 1302 and time1303) shown in FIG. 16 occur to change lens information and generatethree fragments in a moving image file.

Referring to FIG. 17A, the dust position correction data 1502 is addedto the video track 1004 in order to associate dust correction data witheach frame of a moving image. The MP4 file format permits recording dataunique to a system by using an extended BOX with a type ‘uuid’, or usingUser Data Box (‘udta’).

With this mechanism, uuid 1501 is set in the video track of moov or moofof each fragment to write dust position correction data as unique data,as shown in FIG. 17A. The dust position correction data is stored inassociation with each frame until lens information is updated.

The MP4 file format also permits not only recoding ‘uuid’ in the videotracks of moov and moof, but also recording it in parallel to media dataand metadata, like ‘uuid’ 2001 in FIG. 18A. Dust position correctiondata may also be recorded as shown in FIGS. 18A and 18B.

As shown in FIG. 19B, ‘uuid’ 2101 may also be set at the end of a movingimage file. In this case, dust position correction data 2102, 2103, and2104 corresponding to the first, second, and third fragments aredescribed in time series.

Dust position correction data may also be stored as a separate file,like a dust position correction data file 2201 in FIG. 20A. In thiscase, in order to associate the MP4 file 1001 with the dust positioncorrection data file 2201, they need to have the same file name withdifferent extensions, as shown in FIGS. 20A and 20B. Alternatively, theMP4 file 1001 needs to describe the name of the dust position correctiondata file as unique data udta. Needless to say, udta can also recorddust position correction data.

The sequence of dust removal processing for a moving image filecontaining dust position correction data will be explained. A casewherein dust removal processing in FIG. 10 is applied to a moving imagein a separately prepared image processing apparatus will be described.Only a difference when performing the removal processing in FIG. 10 fora moving image file with the file format in FIGS. 17A and 17B will beexplained.

FIG. 21 is a block diagram showing the schematic system configuration ofthe image processing apparatus. A CPU 1601 controls the overall system,and executes a program stored in a primary storage 1602. The primarystorage 1602 is mainly a memory. The primary storage 1602 loads aprogram from a secondary storage 1603, and stores it. The secondarystorage 1603 is, e.g., a hard disk. In general, the primary storage issmaller in capacity than the secondary storage. The secondary storagestores programs, data, and the like which cannot be completely stored inthe primary storage. The secondary storage also stores data which needto be stored for a long time. In the first embodiment, the secondarystorage 1603 stores programs. When executing a program, it is loaded tothe primary storage 1602 and executed by the CPU 1601.

An input device 1604 includes a mouse and keyboard used to control thesystem, and a card reader, scanner, and film scanner necessary to inputimage data. An output device 1605 is, e.g., a monitor or printer. Theapparatus can take other various arrangements, but this is not a gist ofthe present invention and a description thereof will be omitted.

The image processing apparatus incorporates an operating system capableof parallel-executing a plurality of programs. The user can use a GUI(Graphical User Interface) to operate a program running on theapparatus.

FIG. 22 is a view showing the GUI of an image edit program in the imageprocessing apparatus. The window has a close button 1700 and title bar1701. The user ends the program by pressing the close button. The userdesignates a moving image file to be corrected by dragging and droppingit to an image display area 1702. When an image to be corrected isdetermined, the title bar 1701 displays the file name. When the userdesignates a moving image file to be corrected, the image display area1702 displays first frames 2301 of respective fragments side by side asthumbnails. The user clicks and selects the first frame of a displayedfragment. Then, the image display area 1702 displays all frames in thefragment including the first frame side by side as thumbnails. The userclicks and designates a frame to undergo dust removal processing amongall the frames displayed as thumbnails. The frame to be corrected isdisplayed to be fitted in the image display area 1702. When the userpresses an execution button 1703, dust removal processing (to bedescribed later) is executed. The image display area 1702 displays theprocessed image. When the user presses a step execution button 1704, astep of the dust removal processing (to be described later) is executed.At the end of processing all dust regions, the image display area 1702displays the processed image. When the user presses a save button 1705,the target frame is replaced with the processed one to save theresultant moving image file.

The method of designating a frame to be corrected by dust removalprocessing is not limited to this. For example, when the user designatesa moving image file to be corrected, all frames may also be displayedfirst as thumbnails, like the first frames 2301, to prompt the user toselect a frame to be corrected. The user may also designate a fragmentwhile fragments are displayed as thumbnails. In this case, all frames inthe designated fragment are automatically extracted one by one. Asframes to be corrected, the extracted frames sequentially undergo dustremoval processing. Alternatively, the user may designate a moving imagefile. Also in this case, all frames are automatically extracted one byone. As frames to be corrected, the extracted frames sequentiallyundergo dust removal processing.

As described above, the user designates a frame to be corrected by dustremoval processing. This corresponds to step S1801 in FIG. 10. Then, thedust position correction data 1502 added to a fragment containing thedesignated frame to be corrected is obtained. This corresponds to stepS1802. Dust correction data is extracted from the obtained dust positioncorrection data 1502 to perform processing in step S1803. In step S1804,an aperture value and lens pupil position in shooting are obtained fromthe dust position correction data. Step S1805 is executed based on theinformation. In step S1806, correction processing is repetitively doneuntil dust removal is completed (step S1807).

The dust removal processing using a separately prepared image processingapparatus has been described, but the dust removal processing may alsobe done within the digital camera body. When performing dust removalprocessing in the digital camera body, the system control circuit 50performs the same processing as that shown in the flowchart of FIG. 10by executing a dust removal processing program stored in the nonvolatilememory 56. For example, when the user designates the start of dustremoval processing with the four-way selector switch 116, the systemcontrol circuit 50 reads out still image data stored in the memory 30 tothe image processing circuit 20. The image processing circuit 20performs the processing shown in FIG. 10, and executes dust pixelinterpolation processing. Finally, the recording medium 200 records theinterpolation processing result as a new moving image file.

As described above, a moving image file is fragmented at a timing whenlens information such as the aperture value or lens pupil positionchanges. Dust position correction data including lens information anddust position information during shooting is attached to each fragment.This structure obviates the need to pay attention to the correspondencebetween dust position correction data and each frame in a moving imagefile. Dust position correction data is compact data formed from the dustposition, size, and conversion data (aperture value and lens pupilposition information), and does not excessively increase the size ofmedia data such as moov and moof. Interpolation processing is done foronly a region containing pixels designated by dust position correctiondata, so the probability of detection errors can greatly decrease.

(Second Embodiment)

In the first embodiment, a moving image file is fragmented every timelens information is updated upon a lens operation during moving imageshooting. The second embodiment will explain a method of fragmenting amoving image file in accordance with the change amount of lensinformation.

FIG. 23 is a flowchart showing control to fragment a moving image filein accordance with the change amount of lens information. The same stepnumbers as those in FIG. 14 denote the same operations as those in FIG.14, and a difference from FIG. 14 will be mainly explained.

When moving image shooting starts, a new file is created (step S1201),and dust position correction data is created (step S2301). At this time,a memory 52 stores, as P, a lens pupil position in the current lensinformation.

The file stores the created dust position correction data (step S1203).In step S1204, moving image shooting, image processing, and compressionprocessing are performed to store the moving image in the file (stepS1205).

In step S1206, it is checked whether the user has requested the end ofmoving image recording. If the user has not requested the end, it ischecked whether the lens information has been updated (step S1207).

If the lens information has been updated, a lens pupil position in theupdated lens information is compared with P stored in step S2301 tocheck whether the change amount of the lens pupil position is equal toor larger than a predetermined value P0, i.e., the change amount≧P0(step S2302). P0 is an arbitrary value, or a range in which the centercoordinates Di′ are not greatly different between the pupil position Pand an updated pupil position when the center coordinates of dust arecalculated using equation (1) in step S1804 of FIG. 10. When the changeamount≧P0, the center coordinates Di′ greatly change if it is calculatedusing P in step S1804. In dust region determination (step S1901 in FIG.11), none of the conditions are satisfied, and it is determined thatdust does not exist though it actually exists. To prevent this, the lensinformation in the dust position correction data is overwritten to setthe current lens pupil position as P (step S2303).

In step S1209, the fragment position is updated, and the process returnsto step S1203. If the change amount<P0, no fragmentation is executed,and the process returns to step S1204 to perform moving image shooting,image processing, and compression processing.

In the above description, the change amount of the lens pupil positionis obtained. Instead, the change amount of the lens aperture value or acombination of these two change amounts may also be obtained.

As described above, the moving image file is fragmented in accordancewith the change amount of lens information. This can prevent an increasein the size of a moving image file and deterioration of fileaccessibility in playback that are caused by unnecessary filefragmentation.

(Third Embodiment)

In the third embodiment, the contents described with reference to FIGS.1 to 13 are the same as those in the first embodiment.

The operation of the third embodiment will be explained.

(Moving Image Shooting Routine)

An operation in moving image shooting in the third embodiment will bedescribed.

A system control circuit 50 performs this processing by executing amoving image shooting processing program stored in a nonvolatile memory56.

To shoot a moving image, the user needs to change the shooting mode froma still image shooting mode to a moving image shooting mode using a modedial 60 or the like.

When the moving image shooting routine starts, the system controlcircuit 50 operates a quick return mirror 130 shown in FIG. 3 to flip itup and retract it from the imaging optical path. The system controlcircuit 50 opens a shutter 12 to expose an image sensor 14 to objectlight. Image data obtained by exposing the image sensor 14 aresuccessively written at a predetermined rate in a memory 30 serving as aframe buffer. An LCD monitor 417 functions as an electronic viewfinder(EVF) to sequentially display the written image data. In the movingimage shooting mode, it is detected whether the user has pressed themoving image recording start button (e.g., he has pressed a SET button117 in the moving image shooting mode). If so, moving image shootingstarts to sequentially record image data on a recording medium 200 inthe MP4 file format.

When moving image shooting starts upon pressing the moving imagerecording button in the moving image shooting mode, a new file isgenerated first. moov serving as the BOX of metadata of the firstfragment and mdat serving as the BOX of media data are created.

Then, dust position correction data is created. The dust positioncorrection data stores an aperture value and lens pupil positioninformation serving as lens information of a lens used in moving imageshooting, and the dust correction data shown in FIG. 5. A memory 52stores the created dust position correction data. The dust positioncorrection data stored in the memory 52 is read and written in the moovof metadata of the current fragment.

FIG. 24 is a flowchart showing an operation when a lens unit 300 isdriven to zoom during moving image shooting.

In the third embodiment, when the system control circuit 50 detects thata zoom control unit 344 has started zoom driving during moving imageshooting, it performs the following processing.

Upon detecting the start of zoom driving, the system control circuit 50newly creates a fragment (step S1401). The shot moving image data isfragmented to record the fragments.

Then, the system control circuit 50 exposes the image sensor 14 toperform moving image shooting processing. The memory 30 stores thegenerated moving image data. An image processing circuit 20 performsimage processing sequentially for respective frames of the moving imagedata, and the memory 30 records them (step S1402).

The system control circuit 50 receives, from the zoom control unit 344,information representing whether the lens is during zoom driving. Thesystem control circuit 50 determines whether the lens unit 300 is duringzoom driving (zoom operation) (step S1403).

If the system control circuit 50 determines in step S1403 that the lensunit 300 is during zoom driving, it obtains lens information (stepS1404). The lens information includes an aperture value and pupilposition.

The system control circuit 50 determines whether the lens information ofthe current frame that has been obtained in step S1404 has changed fromthat of a previous frame (step S1405).

If the system control circuit 50 determines in step S1405 that the lensinformation has changed, it records the lens information of the currentframe in moof of metadata of the current fragment (step S1406).

If the system control circuit 50 determines in step S1405 that no lensinformation has changed, it performs moving image shooting, imageprocessing, and compression processing without fragmentation, and writesthe moving image data in mdat of the current fragment.

During zoom driving, changed lens information is additionally written inmoof of metadata of one fragment. Simultaneously when recording lensinformation in the header, information on the number of frames or thelike representing the range of frames corresponding to the same lensinformation is also written. The information representing the range offrames corresponding to the same lens information is not limited to thenumber of frames, and may also be another one as far as it can specifylens information and corresponding frames.

If the system control circuit 50 determines in step S1403 that the lensunit 300 is not during zoom driving, it performs fragmentation to newlycreate a fragment, and ends the operation during zoom driving.

The series of processes (steps S1402 to S1406) is repeated until thesystem control circuit 50 determines that zoom driving has ended.

In moving image playback, when converting dust correction parameters instep S1805 in dust removal processing of FIG. 10, lens informationcorresponding to each frame is read out from moof of the fragment toperform dust removal.

The third embodiment provides the following effects.

Since dust correction data is attached to an image in theabove-described manner, this obviates the need to pay attention to thecorrespondence between dust correction image data and shot image data.Dust correction data is compact data formed from the position, size, andconversion data (aperture value and lens pupil position information),and does not excessively increase the size of shot image data.Interpolation processing is done for only a region containing pixelsdesignated by dust correction data, so the probability of detectionerrors can greatly decrease.

During zoom driving of the lens, no fragmentation is executed, andpieces of lens information are recorded in one fragment. Fragmentationis not unnecessarily executed, reducing the data amount.

Since unnecessary fragmentation is not done, the load of moving fileplayback processing decreases.

(Fourth Embodiment)

In the third embodiment, lens information is obtained when recordingeach frame of a moving image during zoom driving of the lens. If thelens information changes, it is recorded.

To the contrary, in the fourth embodiment, lens information is recordedat only the start and end of zoom driving of the lens.

FIG. 25 is a flowchart showing moving image recording processing duringzoom driving in the fourth embodiment.

In the fourth embodiment, when a system control circuit 50 detects thata zoom control unit 344 has started zoom driving during moving imageshooting, it performs the following processing.

Upon detecting that zoom driving has started, the system control circuit50 newly creates a fragment (step S1501).

Then, the system control circuit 50 obtains lens information. As lensinformation at the start of zoom driving, the obtained lens informationis recorded in moof of metadata of the current fragment (step S1502).The lens information includes an aperture value and pupil position.

The system control circuit 50 exposes an image sensor 14 to performmoving image shooting processing. A memory 30 stores the shot movingimage. An image processing circuit 20 performs image processingsequentially for respective frames of the shot moving image, and thememory 30 records them (step S1503).

The system control circuit 50 receives, from the zoom control unit 344,information representing whether the lens is during zoom driving. Thesystem control circuit 50 determines whether the lens is during zoomdriving (step S1504). The processes in steps S1503 and S1504 arerepeated until the system control circuit 50 determines that the lens isnot during zoom driving.

If the system control circuit 50 determines in step S1504 that the lensis not during zoom driving, i.e., zoom driving has ended, it obtainslens information (step S1505). As lens information at the end of zoomdriving, the obtained lens information is recorded in moof of metadataof the current fragment. The lens information includes an aperture valueand pupil position.

The system control circuit 50 performs fragmentation to newly create afragment, and ends the sequence during zoom driving (step S1506).

When converting dust correction parameters in step S1805 in dust removalprocessing of FIG. 10, the pieces of lens information at the start andend of zoom driving are read out from moof of the fragment. Forintermediate frames during zoom driving, lens information isinterpolated based on the difference between the pieces of lensinformation, thereby performing dust removal.

The fourth embodiment can achieve almost the same effects as those ofthe third embodiment.

Further, the fourth embodiment can reduce the data amount because lensinformation is recorded at only the start and end of zoom driving.

(Fifth Embodiment)

The fifth embodiment will be described. In the fifth embodiment, lensinformation is recorded together with a moving image at a predeterminedframe interval during zoom driving of a lens.

FIG. 26 is a flowchart showing moving image recording processing duringzoom driving in the fifth embodiment.

In the fifth embodiment, when a system control circuit 50 detects that azoom control unit 344 has started zoom driving during moving imageshooting, it performs the following processing.

Upon detecting that zoom driving has started, the system control circuit50 newly creates a fragment (step S1601).

Then, the system control circuit 50 obtains lens information, andrecords it in moof of metadata of the current fragment (step S1602). Thelens information includes an aperture value and pupil position.

The system control circuit 50 starts counting frame intervals. Morespecifically, the system control circuit 50 substitutes “1” into thecount value (step S1603).

The system control circuit 50 exposes an image sensor 14 to performmoving image shooting processing. A memory 30 stores the shot movingimage. An image processing circuit 20 performs image processingsequentially for respective frames of the shot moving image, and thememory 30 records them (step S1604).

Upon shooting one frame of the moving image, the system control circuit50 increments, by one, the count value for counting frame intervals(step S1605).

The system control circuit 50 receives, from the zoom control unit 344,information representing whether the lens is during zoom driving.

The system control circuit 50 determines whether the lens is during zoomdriving (step S1606).

The series of processes (steps S1602 to S1607) is repeated until thesystem control circuit 50 determines that the lens is not during zoomdriving.

In step S1607, the system control circuit 50 determines whether thecount value has reached a predetermined frame count (10 frames in FIG.26). The series of processes (steps S1604 to S1607) is repeated untilthe count value reaches the predetermined frame count. If the countvalue reaches the predetermined frame count, the system control circuit50 records lens information in step S1602, starts counting again in stepS1603, and performs the series of processes (steps S1604 to S1607).

If the system control circuit 50 determines in step S1606 that the lensis not during zoom driving, i.e., zoom driving has ended, the systemcontrol circuit 50 performs fragmentation to newly create a fragment instep S1608, and ends the sequence during zoom driving.

When converting dust correction parameters in step S1805 in dust removalprocessing of FIG. 10, pieces of lens information of frames atpredetermined intervals are read out from moov of the fragment. Forintermediate frames for which no lens information is recorded, lensinformation is interpolated based on the difference between precedingand succeeding pieces of lens information, thereby performing dustremoval.

As described above, the fifth embodiment can attain almost the sameeffects as those of the third embodiment. In addition, the fifthembodiment can reduce the data amount because lens information isrecorded at predetermined frame intervals.

(Other Embodiments)

The objects of the embodiments are also achieved by the followingmethod. A storage medium (or recording medium) which stores softwareprogram codes to implement the functions of the above-describedembodiments is supplied to a system or apparatus. The computer (or CPUor MPU) of the system or apparatus reads out and executes the programcodes stored in the storage medium. In this case, the program codes readout from the storage medium implement the functions of theabove-described embodiments. The storage medium that stores the programcodes constitutes the present invention. The functions of theabove-described embodiments are implemented by executing the readoutprogram codes by the computer. The present invention also includes acase wherein the operating system (OS) or the like running on thecomputer executes part or all of actual processing on the basis of theinstructions of the program codes, thereby implementing the functions ofthe above-described embodiments.

The present invention also includes the following case. Morespecifically, the program codes read out from the storage medium arewritten in the memory of a function expansion card inserted into thecomputer or the memory of a function expansion unit connected to thecomputer. The CPU of the function expansion card or function expansionunit executes part or all of actual processing on the basis of theinstructions of the program codes, thereby implementing the functions ofthe above-described embodiments.

When the present invention is applied to the storage medium, the storagemedium stores program codes corresponding to the above-describedprocedures.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2008-148319, filed Jun. 5, 2008, and No. 2008-174954, filed Jul. 3,2008, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An image sensing apparatus comprising: an imagesensing unit having an image sensor which photoelectrically converts anobject image formed via an imaging lens; an optical member which isarranged in front of the image sensor; a foreign substance detectionunit which detects foreign substance information from a foreignsubstance detection image including an image of a foreign substanceadhered to a surface of the optical member, the foreign substanceinformation serving as information on at least a position and size ofthe foreign substance; a lens information obtaining unit which obtainslens information of the imaging lens, the lens information serving asinformation on at least an aperture value and a pupil position of theimaging lens; a memory unit which stores foreign substance positioncorrection data including the foreign substance information and lensinformation obtained when the foreign substance information is detected;and a recording unit which, when shooting a moving image, records movingimage data generated based on image signals successively output from theimage sensing unit, and records the moving image data in associationwith the foreign substance position correction data; wherein when a userchanges the aperture value or the pupil position during moving imageshooting, the lens information obtaining unit updates the lensinformation of the foreign substance position correction data and theforeign substance information of the foreign substance positioncorrection data is not updated, and wherein the recording unit recordsthe moving image data in association with updated foreign substanceposition correction data including the foreign substance informationrecorded when the moving image shooting is started and the updated lensinformation.
 2. The apparatus according to claim 1, wherein when a userchanges the aperture value or the pupil position during moving imageshooting, the lens information obtaining unit updates the lensinformation and the recording unit creates a new fragment, and whereinthe recording unit records the moving image data in association with theupdated foreign substance position correction data in the new fragment.3. A method of controlling an image sensing apparatus including an imagesensing unit having an image sensor which photoelectrically converts anobject image formed via an imaging lens, and an optical member which isarranged in front of the image sensor, the method comprising: a foreignsubstance detection step of detecting foreign substance information froma foreign substance detection image including an image of a foreignsubstance adhered to a surface of the optical member, the foreignsubstance information serving as information on at least a position andsize of the foreign substance; a lens information obtaining step ofobtaining lens information of the imaging lens used to image an object,the lens information serving as information on at least an aperturevalue and a pupil position of the imaging lens; a storing step ofstoring foreign substance position correction data including the foreignsubstance information and lens information obtained when the foreignsubstance information is detected; and a recording step of, whenshooting a moving image, recording moving image data generated based onimage signals successively output from the image sensing unit, andrecording the moving image data in association with the foreignsubstance position correction data; wherein when a user changes theaperture value or the pupil position during moving image shooting, thelens information of the foreign substance position correction data isupdated in the lens information obtaining step and the foreign substanceinformation of the foreign substance position correction data is notupdated, and wherein the moving image data is recorded in associationwith updated foreign substance position correction data including theforeign substance information recorded when the moving image shooting isstarted and updated lens information in the recording step.
 4. Anon-transitory computer-readable storage medium storing a program forcausing a computer to execute a control method defined in claim 3.